Non-incremental classification learning algorithms based on voting feature intervals

Ankara : Department of Computer Engineering and Information Science and the Institute of Engineering and Science of Bilkent University, 1997.Thesis (Master's) -- Bilkent University, 1997.Includes bibliographical references leaves 147-154.Demiröz, GülşenM.S

Towards automatic cost model discovery for combinatorial interaction testing

We present an automated approach for cost model discovery in configuration spaces. Given a configuration space, a quality assurance (QA) task of interest, and a means of measuring the cost of carrying out the QA task, the proposed approach systematically sample the configuration space by using a traditional covering array, carry out the QA task in each of the selected configurations, measure the costs, and fit a generalized linear regression model to the observed costs. The resulting model is then used to estimate the cost of performing the QA task in a possibly previously unseen configuration. The results of our empirical studies conducted on two highly configurable and widely used software systems, strongly support our basic hypothesis that the proposed approach can efficiently and effectively discover reliable cost models

Moving forward with combinatorial interaction testing

Combinatorial interaction testing (CIT) is an efficient and effective method of detecting failures that are caused by the interactions of various system input parameters. In this paper, we discuss CIT, point out some of the difficulties of applying it in practice, and highlight some recent advances that have improved CIT’s applicability to modern systems. We also provide a roadmap for future research and directions; one that we hope will lead to new CIT research and to higher quality testing of industrial systems

Generating cost-aware covering arrays for free

Software systems generally have a large number of configurable options interacting with each other. Such systems are more likely to be prone to errors, crashes, and faulty executions that are usually caused by option interactions. To avoid such errors, testing all possible configurations during the development phase is usually not feasible, since the number of all possible configurations is exponential in the order of number of options. A t-way covering array (CA) is a 2-dimensional combinatorial object that helps to efficiently cover all t-length option interactions of the system under test. Generating a CA with a small number of configurations is important to shorten the testing phase. However, the testing cost (e.g. the testing time) may differ from one configuration to another. Currently, most sequential tools can generate optimum CAs in terms of number of configurations, but they are not cost-aware, i.e., they cannot handle the varying costs of configurations. In this work, we implement a parallel, cost-aware CA-generation tool based on a sequential tool, Jenny, to generate lower-cost CAs faster. Experimental results show that our cost-aware CA construction approach can generate 32% and 21% lower cost CAs on average for t = 2 and t = 3, respectively, compared to state-of-the-art CA-generation tools. Moreover, the cost-awareness comes for free, i.e., we speed up our algorithm by leveraging parallel computation. The cost models and cost reduction techniques we propose could also be adapted for other existing CA generation tools

Generating cost-aware covering arrays for free

Software systems generally have a large number of configurable options interacting with each other. Such systems are more likely to be prone to errors, crashes, and faulty executions that are usually caused by option interactions. To avoid such errors, testing all possible configurations during the development phase is usually not feasible, since the number of all possible configurations is exponential in the order of number of options. A t-way covering array (CA) is a 2-dimensional combinatorial object that helps to efficiently cover all t-length option interactions of the system under test. Generating a CA with a small number of configurations is important to shorten the testing phase. However, the testing cost (e.g. the testing time) may differ from one configuration to another. Currently, most sequential tools can generate optimum CAs in terms of number of configurations, but they are not cost-aware, i.e., they cannot handle the varying costs of configurations. In this work, we implement a parallel, cost-aware CA-generation tool based on a sequential tool, Jenny, to generate lower-cost CAs faster. Experimental results show that our cost-aware CA construction approach can generate 32% and 21% lower cost CAs on average for t = 2 and t = 3, respectively, compared to state-of-the-art CA-generation tools. Moreover, the cost-awareness comes for free, i.e., we speed up our algorithm by leveraging parallel computation. The cost models and cost reduction techniques we propose could also be adapted for other existing CA generation tools

Cost-Aware combinatorial interaction testing

Exhaustive testing of highly configurable software systems is generally infeasible as the number of possible configurations grows exponentially with the number of configuration options. Combinatorial Interaction Testing approaches systematically sample the configuration spaces and test only the selected configurations. The sampling is typically carried out by computing a combinatorial object, called a t-way covering array. Given a configuration space model that includes a set of configuration options, each of which takes a value from a discrete domain, together with inter-option constraints (if any), which invalidate certain combinations of option values, a t-way covering array is a set of valid configurations in which each valid combination of option values for every combination of t options appears at least once. The basic justification for using covering arrays is that they can e ectively exercise the system behaviors caused by the values of t or fewer options. Consequently, covering arrays have been widely used for testing in many domains, including the systematic testing of configurable systems, input parameter spaces, graphical user interfaces, network protocols, and software product lines. To reduce the actual cost of testing, covering arrays aim to reduce the number of configurations selected. By doing so, they implicitly assume that the cost of testing each configuration is the same. In this thesis, we, however, empirically demonstrate that, in practice, the cost often varies from one configuration to another and that when the cost varies, reducing the number of configurations is not necessarily the same as reducing the actual cost of testing. To overcome this issue, we first define a novel combinatorial object for testing called, a cost-aware covering array, which takes the cost of testing into account when computing interaction test suites. In a nutshell, given a configuration space model, enhanced with a cost function, which models the actual cost of testing at the level of option value combinations, a t-way cost-aware covering array is a standard t-way covering array that “minimizes” the given cost function, rather than the number of configurations selected. We then develop specialized construction approaches to turn two di erent types of existing covering arrays, namely standard covering arrays and test case-aware covering arrays, into cost-aware interaction test suites. The results of our experiments conducted on large, widely-used, and highly-configurable software systems strongly suggest that costaware covering arrays can significantly reduce the actual cost of testing without adversely a ecting the coverage properties, compared to existing covering arrays. One thing we observe in all these studies is that it could be di cult for practitioners to develop accurate and precise cost functions. The opportunity to further reduce the testing cost decreases as the cost function is less accurate or less precise. To address this, we develop e cient and e ective approaches to automatically discover the cost model in configuration spaces